System and method for association assisted establishment of scattering configuration in scattering processing

ABSTRACT

A system for association-based scattering processing includes a spatial light modulator configured to modulate one or more of phase and amplitude of light irradiated from a light source to a target object. Additionally, the system includes processing circuitry configured to evaluate a field distribution for one localized illumination, induce a set of field distributions for a plurality of localized illuminations based on the field distribution for the one localized illumination, and apply the set of field distributions to the spatial light modulator, scanning a plurality of localized illuminations on the target object.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims the benefit of U.S. Provisional Application No.62/797,363, filed Jan. 28, 2019, and U.S. Provisional Application No.62/797,366, filed Jan. 28, 2019, which are incorporated herein byreference in their entirety. Additionally, this application is relatedto attorney docket numbers 13060US01, 13062US01, and 13241WO01, whichare incorporated herein by reference in their entirety.

BACKGROUND

A severe scattering medium can cause issues for an image capturingdevice trying to image a target in or through the severe scatteringmedium. When evaluating a severe scattering medium (e.g., fog), theevaluation of the scattering medium must be finished within thedecorrelation time of the scattering medium. For example, thedecorrelation time for fog is about 5 ms. In other words, thedecorrelation time describes the amount of time the fog remainsunchanged. Accordingly, the time it takes to evaluate the scatteringproperties of fog must be less than 5 ms. However, because fog moves andchanges so quickly, evaluating a scattering medium like fog usingtraditional methods is impractical.

The “background” description provided herein is for the purpose ofgenerally presenting the context of the disclosure. Work of thepresently named inventors, to the extent it is described in thisbackground section, as well as aspects of the description which may nototherwise qualify as prior art at the time of filing, are neitherexpressly or impliedly admitted as prior art against the presentinvention.

SUMMARY

According to aspects of the disclosed subject matter, a system forassociation-based scattering processing includes a spatial lightmodulator configured to modulate one or more of phase and amplitude oflight irradiated from a light source to a target object. Additionally,the system includes processing circuitry configured to evaluate a fielddistribution for one localized illumination, induce a set of fielddistributions for a plurality of localized illuminations based on thefield distribution for the one localized illumination, and apply the setof field distributions to the spatial light modulator, scanning aplurality of localized illuminations on the target object.

The foregoing paragraphs have been provided by way of generalintroduction, and are not intended to limit the scope of the followingclaims. The described embodiments, together with further advantages,will be best understood by reference to the following detaileddescription taken in conjunction with the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

A more complete appreciation of the disclosure and many of the attendantadvantages thereof will be readily obtained as the same becomes betterunderstood by reference to the following detailed description whenconsidered in connection with the accompanying drawings, wherein:

FIG. 1 illustrates an exemplary scattering processing system accordingto one or more aspects of the disclosed subject matter;

FIG. 2 illustrates an exemplary workflow for association assistedscattering processing according to one or more aspects of the disclosedsubject matter;

FIG. 3 is an exemplary workflow for evaluating a low dimensiontransmission matrix for fixed focusing according to one or more aspectsof the disclosed subject matter;

FIG. 4 illustrates induction of higher dimension TMs via associationamong low and high dimension TMs according to one or more aspects of thedisclosed subject matter;

FIG. 5A illustrates induction of optical field for neighboring localizedillumination via association among optical fields according to one ormore aspects of the disclosed subject matter;

FIG. 5B illustrates exemplary phase associations according to one ormore aspects of the disclosed subject matter;

FIG. 5C illustrates exemplary amplitude associations according to one ormore aspects of the disclosed subject matter;

FIG. 5D illustrates the result of analyzing FIG. 5B by binary evolutionwhich ranges from 0 to π according to one or more aspects of thedisclosed subject matter;

FIG. 5E illustrates a graph showing a periodic like (distance betweenphase stripes) change of the phase differences for the weak scatteringmedium according to one or more aspects of the disclosed subject matter;

FIG. 6 illustrates scanning of focused illuminations by applying theinduced optical fields on the modulator according to one or more aspectsof the disclosed subject matter;

FIG. 7A is an algorithmic flow chart of method for imaging a targetobject in different conditions according to one or more aspects of thedisclosed subject matter;

FIG. 7B is an algorithmic flow chart of a method for imaging a targetobject through a scattering medium according to one or more aspects ofthe disclosed subject matter; and

FIG. 8 is a hardware block diagram of a server according to one or moreexemplary aspects of the disclosed subject matter.

DETAILED DESCRIPTION

The description set forth below in connection with the appended drawingsis intended as a description of various embodiments of the disclosedsubject matter and is not necessarily intended to represent the onlyembodiment(s). In certain instances, the description includes specificdetails for the purpose of providing an understanding of the disclosedsubject matter. However, it will be apparent to those skilled in the artthat embodiments may be practiced without these specific details. Insome instances, well-known structures and components may be shown inblock diagram form in order to avoid obscuring the concepts of thedisclosed subject matter.

Reference throughout the specification to “one embodiment” or “anembodiment” means that a particular feature, structure, characteristic,operation, or function described in connection with an embodiment isincluded in at least one embodiment of the disclosed subject matter.Thus, any appearance of the phrases “in one embodiment” or “in anembodiment” in the specification is not necessarily referring to thesame embodiment. Further, the particular features, structures,characteristics, operations, or functions may be combined in anysuitable manner in one or more embodiments. Further, it is intended thatembodiments of the disclosed subject matter can and do covermodifications and variations of the described embodiments.

It must be noted that, as used in the specification and the appendedclaims, the singular forms “a,” “an,” and “the” include plural referentsunless the context clearly dictates otherwise. That is, unless clearlyspecified otherwise, as used herein the words “a” and “an” and the likecarry the meaning of “one or more.” Additionally, it is to be understoodthat terms such as “left,” “right,” “top,” “bottom,” “front,” “rear,”“side,” “height,” “length,” “width,” “upper,” “lower,” “interior,”“exterior,” “inner,” “outer,” and the like that may be used herein,merely describe points of reference and do not necessarily limitembodiments of the disclosed subject matter to any particularorientation or configuration. Furthermore, terms such as “first,”“second,” “third,” etc., merely identify one of a number of portions,components, points of reference, operations and/or functions asdescribed herein, and likewise do not necessarily limit embodiments ofthe disclosed subject matter to any particular configuration ororientation.

In severe scattering processing, severe scattering mediums can includebiological tissue, heavy fog, rain, snow, and the like. In these severescattering mediums, most of the incident light is scattered, and, as aresult, only a small amount of incident light reaches the targets inside(or through) the scattering medium. And, even less light is reflectedfrom the target and detected. As a result, a sensitive detector isneeded to capture the direct incident and reflected light, calledballistic light. However, the low signal to noise ratio (SNR) hindersthe capability to retrieve the information from the target. Therefore,only shallow depths can be reached inside the scattering medium.

For the light to reach the target deeper inside the scattering medium,there are two ways to increase SNR. A first option is to concentrate theillumination (i.e., focus the light) on the target and scan thelocalized illuminations on the target. The other is to divert thescattered light back on the targets. The former approach is themanipulation of the ballistic light, while the latter is the modulationof the scattered light.

To make use of the scattered light, the scattering medium is evaluatedfirst, then the incident light can be modulated to divert the scatteredlight on the target inside or behind the scattering medium. For thefast-varying scattering media, like fog, rain, or biological tissue,decorrelation time is used to describe the time during which thescattering property stays unchanged. In order to divert the scatteredlight, the evaluation of the scattering medium must be finished withinthe decorrelation time. The decorrelation time of biological tissue isusually 50-2000 ms, while for fog, it is usually 5 ms. When thescattering medium is evaluated, the output of the optical field afterthe modulation should be defined beforehand. For example, a requiredfocus at a specific position on the target inside (or through) thescattering medium can be achieved by measuring the output intensities atthis position while several spatial modulated incident light inputs areilluminated into the scattering medium. By solving the establishedequations relating to the inputs and outputs, a matrix which describesthe scattering property of the scattering medium can be acquired.Conjugation of the matrix can give the optical field distribution (phaseor amplitude), and, when applied on the spatial light modulator, therequired focused illumination after the scattering medium can beachieved. The efficiency, or the enhancement of the focusedillumination, is proportional to the dimension of the incidentmodulation (i.e., the number of the modulations on the incident light).A higher number of incident modulation used corresponds to a higherefficiency of the focused illumination that can be achieved, but moretime is needed to evaluate the scattering medium. And much more time isneeded to evaluate the scattering medium for multiple focusedilluminations.

Typically, a Digital Micromirror Device (DMD) is used as the opticalfield modulator, which has the ability of operating the micromirrorcurrently with 22 kHz, corresponding to 44 μs to display a pattern onit. For an 8×8 incident field modulation and one localized illuminationas output, the time to finish the evaluation is about 64×3×44=˜8 ms.Here, the “3” stands for three-time interferences from three differentreference beams, for example, 0, pi/2, pi, for each input field tocalculate the output optical field because the detector (camera) canonly record intensity, which is the conjugated square of the outputoptical field. 44 μs stands for the time needed to display a pattern onDMD. The time to evaluate the 32×32 dimension modulation with onelocalized illumination output is about 32×32×3×44=˜135 ms. For 64×64, itwould be 540 ms. To get a focus illumination with a high enhancement,the high dimension incident modulation is needed. Additionally, theevaluation of a high number of focused illuminations is also highlytime-consuming. Previously, no approach could be employed to get therequired optical fields to achieve the scanning of the focusedilluminations on the fast-varying scattering media, such as biologicaltissues, fog, etc. However, one or more aspects of the disclosed subjectmatter describe techniques for quickly evaluating the fast-varyingscattering media, like fog, to manipulate the scattered light for betterimaging of the targets inside the scattering media.

Referring now to the drawings, wherein like reference numerals designateidentical or corresponding parts throughout the several views:

FIG. 1 illustrates an exemplary scattering processing system 100 (hereinreferred to as the system 100) according to one or more aspects of thedisclosed subject matter. As will be discussed in more detail later, oneor more methods according to various embodiments of the disclosedsubject matter can be implemented using the system 100 or portionsthereof. Put another way, system 100, or portions thereof, can performthe functions or operations described herein regarding the variousmethods or portions thereof (including those implemented using anon-transitory computer-readable medium storing a program that, whenexecuted, configures or causes a computer to perform or causeperformance of the described method(s) or portions thereof).

The system 100 can include a light source 105, a spatial light modulator110, a detector 115, a database 120, and processing circuitry 130 (whichcan include internal and/or external memory). In one or more aspects ofthe disclosed subject matter, the light source 105, the spatial lightmodulator 110, the detector 115, the database 120, and the processingcircuitry 130 can be implemented in an apparatus 102. The apparatus 102can represent various apparatuses that perform imaging through ascattering medium. For example, the apparatus 102 can be an autonomousvehicle where the headlights can adapt to fog (and/or other scatteringmedia) using the scattering processing system 100. As a result, theautonomous vehicle can more clearly image the road ahead in and throughthe fog, thus improving the autonomous driving capability in thescattering medium. Additionally, in another embodiment, the apparatus102 can be an apparatus for real time in vivo imaging through biologicaltissue, in which the system 100 can overcome the dynamic scatteringprocessing caused by the physiological environment. Further, theaforementioned components can be electrically connected or in electricalor electronic communication with each other as diagrammaticallyrepresented by FIG. 1, for example.

Generally speaking, the system 100 can quickly evaluate a fast-varyingscattering medium (e.g., fog) to improve visualization in and/or throughthe fast-varying scattering medium. For example, the system 100including the light source 105, the spatial light modulator 110, thedetector 115, the database 120, and the processing circuitry 130 can beconfigured to, via the processing circuitry, project, via the lightsource, illumination based on a predetermined illumination profile.Next, the spatial light modulator can modulate an optical field of theprojected illumination based on the illumination profile. Additionally,the detector can detect backscattered illumination corresponding to afirst portion of the projected illumination backscattered from ascattering medium and reflected illumination corresponding to a secondportion of the projected illumination reflected from a target objectlocated in or through the scattering medium. Finally, the processingcircuitry can induce the illumination profile based on one localizedillumination, wherein other localized illuminations can be formed atdifferent positions through the scattering medium based on anassociation relationship among the optical fields for the focusedilluminations in or through the scattering medium.

More specifically, to improve the visualization in the case of severeweather conditions, like fog, rain, snow, and/or heavy pollution, thesystem 100 can quickly establish the overall scattering properties of ascattering medium from a limited knowledge about the scattering medium.The scattering properties can be established based on associationswithin the scattering processing. For example, by associatingtransmission matrices in different dimensions of the same system and byassociating optical fields in the same dimension of the transmissionmatrix, the optical localized illuminations and scanning configurationsystem inside or through the scattering medium can be quicklyestablished.

For example, the system 100 can calculate the transmission matrix of thescattering medium with lower dimension incidence modulation for onefocused output (e.g., 8×8 incidence modulations). In this way, theevaluation of the scattering medium can be finished in the shortestamount of time. Next, the system 100 can induce the transmission matrixof a higher incidence dimension by applying an association relationshipbetween the transmission matrices of different dimensions. In this way,the higher dimension transmission matrix describing the scattering mediacan be obtained without any experimental evaluation processing, whichsaves a significant amount of time. Additionally, the optical fielddistribution (phase or amplitude) extracted from the transmission matrixwith higher dimension can form localized illuminations with higherenhancement.

Then, the system 100 can induce the optical field (phase or amplitude)distributions forming the focused illuminations at neighboring positionsbased on the associations of the optical fields corresponding to thefocused illuminations for the same dimension of the transmission matrix.Finally, the system 100 can perform fast localized (i.e., focused)scanning with high enhancement quickly enough to image through afast-changing dynamic scattering medium (e.g., fog).

In other words, instead of experimental evaluations of the scatteringmedium for larger dimension matrix and multiple focused illuminations,which would take huge amount of time and make it impossible to form theimage of the fast evolving medium with confocal configurations, thesystem 100 uses associations between transmission matrices withdifferent dimension and associations between optical fieldsdistributions to form the localized illuminations in the same dimensionof transmission matrix.

The light source 105 can represent one or more light sources in thesystem 100. In one or more aspects of the disclosed subject matter, thelight source 105 can be the headlight of a vehicle. For example, if theapparatus 102 is a vehicle, the system 100 can be configured to modulatethe output of the one or more headlights of the vehicle to improvevisualization through various scattering media.

The spatial light modulator 110 can represent one or more spatial lightmodulators in the system 100. A spatial light modulator can modulate theintensity of a light beam. Additionally, a spatial light modulator canmodulate the phase or both the phase and amplitude simultaneously. Forexample, the spatial light modulator 110 can be a Digital MicromirrorDevice (DMD) which can include a plurality of micromirrors arranged in amatrix. In one or more aspects of the disclosed subject matter, thespatial light modulator 110 can modulate the output of the light source105.

The detector 115 can represent one or more detectors in the system 100.In one or more aspects of the disclosed subject matter, the detector 115can be an imaging device. Although other types of detectors can becontemplated, imaging device and detector can be used interchangeablyherein. For example, if the apparatus 102 is an autonomous vehicle, thedetector 115 can represent one or more imaging devices used forautonomous operation of the vehicle. Accordingly, the system 100 canimprove the imaging device's ability to operate in various scatteringmedia.

The database 120 can represent one or more databases in the system 100.The database 120 can be configured to store various information foroperation of the system 100. For example, the database 120 can storeinformation for a plurality of transmission matrices, information forphase patterns, and information for phase differences as furtherdescribed herein. Alternatively, or additionally, the database 120 canrepresent a memory of the processing circuitry 130, for example.

The processing circuitry 130 can carry out instructions to perform orcause performance of various functions, operations, steps, or processesof the system 100. In other words, the processor/processing circuitry130 can be configured to receive output from and transmit instructionsto the one or more other components in the system 100 to operate thesystem 100 to improve visualization through various scattering media.

FIG. 2 illustrates an exemplary workflow 200 for association assistedscattering processing according to one or more aspects of the disclosedsubject matter. The associations allow the system 100 to quickly assessthe scattering medium for adaptive applications as further describedherein.

A direct evaluation of a severe scattering medium (e.g., fog) for highdimension incidence modulation and multiple focusing outputs cannot beaccomplished within the decorrelation time. In other words, because asevere scattering medium like fog moves and changes so quickly (i.e., 5ms decorrelation time), the direct evaluation for high dimensionincidence modulation and multiple focusing outputs cannot be completedin less than 5 ms. Accordingly, rather than a direct evaluation of thescattering medium for high dimension incidence modulation and multiplefocusing outputs, the system 100 is configured to use an indirectapproach which starts with using a low dimension incidence modulationand only one focusing output to evaluate the scattering medium.

Generally, a first step S205 of the workflow 200 includes evaluation ofthe scattering medium 230 for a low dimension incidence modulation andone focusing output 235. The TM is the relation between the input fielddistribution (on the modulator) and the output field distribution. Thefield distribution (phase or amplitude) corresponding to one localizedillumination is the conjugation of the corresponding TM. Accordingly, inS205, the system 100 calculates a low dimension transmission matrix (TM)240 for the scattering medium 230 by using the low dimension incidencemodulation. Then, in S205, the system 100 calculates a fielddistribution in low dimension for only one focusing output 235 using theTM 240. Next, in S210, the system 100 can induce a higher dimensionfield distribution for one localized illumination based on the lowdimension field distribution calculated in S205. The high dimensionfield distribution can be induced from the low dimension fielddistribution based on an association relationship among the transmissionmatrices with different dimensions. In other words, the associationrelationship can be applied to determine a TM with a high dimension ofmodulation (e.g., high dimension TM 245).

In S215, the system 100 can induce an optical field for neighboringfocusing based on an association among optical fields. Additionally, theoptical field distribution with the high dimension of modulation can beused to form a focused illumination with high enhancement. Theassociation relationship among the field distributions corresponding tothe focused illuminations can be applied to induce the incident fielddistributions for the neighboring focused illuminations. Finally, inS220, the system 100 can scan (e.g., received at the detector 115) thefocused illuminations by applying the induced optical fields on themodulator. More specifically, by applying the acquired incident opticalfields on the spatial light modulator (e.g., spatial light modulator110), the imaging device (e.g., the detector 115) can received thescanned focused illuminations on the targets and get a confocal image ofthe target for better visibility. Steps S205, S210, S215, and S220 aredescribed in more detail in FIGS. 3, 4, 5, and 6, respectively.

FIG. 3 is an exemplary workflow 300 for evaluating a low dimensiontransmission matrix for fixed focusing (e.g., S205) according to one ormore aspects of the disclosed subject matter.

In order to evaluate the scattering medium 230, a transmission matrixcan be used to describe the relationship between the input optical fieldinto the scattering medium and the output optical field after thescattering medium. In other words, the random inputs of optical fieldscan be used to evaluate the transmission matrix. Because the correlatedinputs might be used, the number of the inputs are usually higher thanthe unknown parameters in the transmission matrix. For example, thenumber of inputs would be higher than MxN for the (M,N) dimensiontransmission matrix. In order to eliminate the uncertain number of theinputs, a Hadamard matrix can be used (S305). A Hadamard matrix is amatrix where each row is independent and the Hadamard matrix can be usedto generate the independent inputs.

Usually, three or four inputs on the modulator (e.g., the spatial lightmodulator 110) are created by each row in the Hadamard matrix in orderto evaluate the transmission matrix of the scattering medium. Therefore,the number of the inputs is more than the number of the row in theHadamard matrix. As shown in S310, each row in the Hadamard matrix isconverted to two-dimension distributions, I_(in), for the realization ofthe incidence on the modulator. In one aspect, the modulator can be thespatial light modulator 110. Additionally, in one or more aspects of thedisclosed subject matter, the modulator can be a digital micromirrordevice (DMD).

In S5315, for each incidence from the modulator, an intensity value onthe detector after the scattering medium is acquired as I_(out) by thedetector (e.g., detector 115). After acquiring all the outputscorresponding to the inputs in S5315, the equations to relate the inputsand the outputs can be established regarding the unknown parameters inthe transmission matrix as shown in the Equation 1 below.

$\begin{matrix}{\begin{bmatrix}I_{out_{1}} \\I_{out_{2}} \\\vdots \\I_{out_{m}}\end{bmatrix} = {\begin{bmatrix}{TM}_{11} & {TM}_{12} & \ldots & {TM}_{1\; n} \\{TM}_{21} & T_{22} & \ldots & {TM}_{2\; n} \\\vdots & \vdots & \vdots & \vdots \\{TM}_{m\; 1} & {TM}_{m\; 2} & \ldots & {TM}_{mn}\end{bmatrix}\begin{bmatrix}I_{in_{1}} \\I_{in_{2}} \\\vdots \\I_{in_{n}}\end{bmatrix}}} & \left( {{Equation}\mspace{14mu} 1} \right)\end{matrix}$

In S320, the transmission matrix of the scattering medium can beevaluated by solving the above equations. Finally, in S325, fielddistribution for localized illumination behind the scattering medium canbe acquired. More specifically, conjugation of the evaluatedtwo-dimension distribution is the field (phase or amplitude) modulationto form the localized illumination through the scattering medium.

The time spent on each step shown in FIG. 3 can depend on theconfiguration and the equipment used in the step. Time to generate theHadamard matrix and formulate the incident modulations for the modulatorcan be negligible, which can be predefined and managed based on a FieldProgramable Gate Array (FPGA). On the other hand, the time to displaythe incident modulation patterns on the modulator affect the overalloperation time due to the large amount of data transferring involved.For example, critical factors needed to establish the base for theevaluation of the required dimension of the transmission matrix includethe clock frequency of the modulator, the time to transfer the values onthe pixels, the number of pixels of the modulator, and the number of theincident modulations needed. Additionally, the time spent on thedetection can be negligible when a fast detector is chosen, and the timefor evaluating the TM for solving the equations can also be negligiblebecause parallel calculation is involved in fast operation boards.

Therefore, when the evaluation configurations and the equipment arepredefined, the main factor to affect the evaluation time is the numberof the incident modulations, which is defined by the dimension of therequired transmission matrix. For a lower dimension TM, less time isneeded to evaluate the TM. However, the incident field distribution withlow dimension acquired from the low dimension TM only generate lowenhancement at the localized illumination. Accordingly, after takingadvantage of the speed of evaluating a low dimension TM, theassociations among the TMs can be applied to induce the higher dimensionTMs, for example.

FIG. 4 illustrates induction of higher dimension TMs via associationamong low and high dimension TMs (e.g., S210) according to one or moreaspects of the disclosed subject matter. For example, a relationship 405(e.g., f(l,p,m,n)) between the low dimension TM 240 (e.g., TM_((m,n)))and the high dimension TM 245 (e.g., TM_((l,p))) exists because both TMsdescribe the same scattering medium (e.g., scattering medium 230). Therelationship between the low dimension TM 240 and the high dimension TM245 (as well as the relationship between the optimized fielddistributions corresponding to the same localized illumination position)can be established via different approaches. For example, therelationships can be established via neural networks, deep learning, andthe like. Once the association (i.e., relationship) is established, thehigh dimension TM 245 can be induced based on the low dimension TM 240via the established association between the low dimension TM 240 and thehigh dimension TM 245. Therefore, a field distribution in high dimensionfor one localized illumination can be acquired by using the fielddistribution in low dimension for the localized illumination and theassociation among low and high dimension TMs.

FIG. 5A illustrates induction of optical fields for neighboringfocusing, or localized illuminations, via association among opticalfields (e.g., S215) according to one or more aspects of the disclosedsubject matter. As has been described herein, for the determinedlocalized illumination position, the high dimension TM is induced fromthe low dimension TM. By applying the associations among the TMs fordifferent localized illumination positions, as established viacalculations as described further herein, the TMs or the spatial field(phase, amplitude) distributions for the neighboring positions can beinduced. Additionally, the optical phase distributions among theneighboring illumination positions have global associations, which aredistinct from the local relation for the memory effect. The globalassociations are distinct from the local relation because the globalassociations refer to the relationship between the phase distributionsfor the different localized illuminations which is not affected by justlocal factors like limited displacement or spatial frequency ranges, forexample. In other words, “global” can describe that the associationrelationship is across the scattering medium and not a local effect likethe memory effect. Further, global associations exist for the amplitudemodulations corresponding to the neighboring localized illuminationpositions. Accordingly, based on these association relationships, theinduced optical field distributions 505 (or the induced TMs) for thedifferent focusing positions can easily be established without anyexperimental operations, which saves a significant amount of time whenestablishing the spatial field distributions to scan the target area.

FIG. 5B and FIG. 5C illustrate simulation and experimentation supportingphase associations (e.g., associations among phase information betweentransmission matrices) and amplitude associations, respectively. Insummary, by combining the simulation and experimentation results, whenusing wave front shaping to compensate the diffuse light through ascattering medium to form a localized illumination on a target plane,the phase (or amplitude) information between multiple optimizingtransmission matrices is not independent. In fact, it can be representedmathematically. The results show that the phase (or amplitude)difference of different optimizing transmission matrices is globalwithin the area of detection, rather than a local effect, by comparingthe intensity of localized illuminations and phase (or amplitude)distribution of optimizing transmission matrices. Further, the resultsdemonstrate that the difference between optimizing transmission matricesand linear phase gradience is relative and can be predicted by equationbased on a known starting point of linear phase variance.

FIG. 5B illustrates an exemplary phase comparison by simulation 510 andexperimental results 512 according to one or more aspects of thedisclosed subject matter. Additionally, the experimental results 512include extracted localized illuminations and association among opticalphase distributions for the localized illuminations. In this example,the extracted localized illuminations are 40×40 localized illuminationsformed by employing eigenvectors from TM evaluation. In one aspect, acenter can be selected for a 100×100 pixel area on a CCD andconcatenated in CCD area, for example. The experimental results 512 alsoillustrate phase differences among the eigenvectors as optical phasescorresponding to the localized illuminations and a reference (e.g.,center). In other words, the phase differences can be calculated basedon one phase distribution pattern as a reference. In this case, thephase distribution of a center localized illumination is selected as thereference. The value of phase distribution is in a range of (−π, π) andthe phase distribution was found to be periodic. The concentric circlesappeared in the phase difference pattern, and the distance between theconcentric circles become increasingly thinner farther away from thecenter.

FIG. 5C illustrates an exemplary amplitude associations 514 andexperimental verification 515 according to one or more aspects of thedisclosed subject matter. More specifically, the amplitude simulation514 illustrates that associations exist in the amplitude modulation forthe evaluation of TMs of the scattering matrices. The simulation of theassociations of the amplitude modulation from a 64×64 transmissionmatrix shows the difference of the acquired eigenvectors of amplituderegarding the selected localized illumination. Additionally, theexperimental results 515 include extracted localized illuminations andassociation among amplitude distribution for the localizedilluminations. The experimental results 515 illustrate experimentalverification of the associations among the evaluated eigenvectors ofamplitude, acquired from 32×32 transmission matrix by amplitude onlywave front shaping experiment.

FIG. 5D illustrates the result of analyzing FIG. 5B by binary evolutionwhich ranges from 0 to π. More specifically, FIG. 5D illustrates thetrend of phase stripes for three directions are highly similar as thedifference between distances of circles becomes thinner. Though thestripes on the edge are visual illusion that might misguide the result,it is preferably removed by modulating the resolution of screen. Thewidth of phase stripes (distance between phase stripes) decreases in thedirection of 0°, 45°, and 90°, respectively. The phase stripes in thesethree directions are visually similar with the π area and 0 areaalternate dominates.

FIG. 5E illustrates a graph showing a periodic like (distance betweenphase stripes) change of the phase differences for the weak scatteringmedium, in which the horizontal axis represents stripe number and thevertical axis represents width of phase stripes. The periodicity of thewidth of stripes is decreased as summation of two exponential functionsin three directions (0°, 45°, and 90°). Approximate curve for theaverage value of width for three directions is expressed by Equation 2.

T _(M)=1.351−e ^((−0.6294M))+0.295−e ^((−0.08412M))  (Equation 2)

M is the order of phase difference pattern and the distance betweenphase stripes is represented by the number of pixels of phase differencepatterns. From the equation, it can be predicted that the periodicitywill be saturated.

Based on these association relationships, the field distributions forthe different localized illuminations could be easily induced withoutany experimental operations. The information of these associationrelationships (phase differences corresponding to the set of fielddistributions for localized illuminations) are abstracted from aplurality of phase patterns for generating the plurality of localizedilluminations at the spatial light modulator and can be measured inadvance and stored in memory.

FIG. 6 illustrates scanning of localized illuminations by applying theinduced optical fields on the modulator according to one or more aspectsof the disclosed subject matter. For example, the induced fielddistributions 505 (or TMs) are displayed on the modulator 110, whichmodulates the output field to form localized illuminations 605 on thetargets as illustrated in a display 610 of the detector 115. Thescanning of the localized illuminations 605 can form the two-dimensionscan around the initial localized illuminations. The associations of theoptical fields 505 (or TMs) from the initial localized illumination showthe global relationship, which means for any points away from theinitial point, the corresponding field distributions can be induced fromthe initial field distribution. However, due to the limited spatialresolution of the modulator 110, or the limited size of the pixel area,the induced field distribution for the localized illuminations greaterthan a predetermined distance from the initial position may not bespatially resolved by the modulator 110. The solution to this issuewould be to use the induced field distribution to act as the new initiallocalized illumination position, and from that point the fielddistributions corresponding to its neighboring points are generated.This process can be repeated to form the field distributions for all thelocalized illuminations to scan the entire target area (verticallyand/or horizontally). Accordingly, based on the images of the localizedilluminations, the image of the target can be formed via the confocalimage processing.

Additionally, in another embodiment, the advantages of the one or moreaspects of the disclosed subject matter can be adapted for other imagingdevices or systems imaging through scattering media. For example, theapparatus 102 can be a confocal imaging system for real time in vivofluorescent imaging through biological tissue. Accordingly, the dynamicscattering processing caused by the physiological environments such asblood flow, pulse or body liquid, can be overcome by diverting thescattered light back into the target for high contrast images. Thenature of the spatial modulation of the scattering has the capability toform diffraction limited focus. Therefore, the system 100 can be used toform super resolution fluorescent images. Particularly, phaseconjugation with a FPGA program can be employed to retrieve the opticalphase through the biological tissue within 3 ms. Similarly, using thephase conjugation to capture the optical phase (or TM) for a focusedillumination can achieve the speed required to evaluate the scatteringprocessing of a fast-varying scattering medium like fog, which can makeuse of the scattering by the fog to significantly improve thevisualization through the fog and revolutionize the lighting designindustry.

FIG. 7A is an algorithmic flow chart of method for imaging a targetobject in different conditions according to one or more aspects of thedisclosed subject matter.

In S705, it can be determined if the system 100 detects a scatteringmedium. For example, the system 100 may detect that a vehicle isapproaching or traveling through a scattering medium. For the followingdescription, the scatter medium can be fog. However, it should beappreciated that the process can be applied to any scattering mediumthat has been mentioned herein. In one aspect, the imaging device (i.e.,the detector 115) can detect a presence of the fog based on a decreasein visibility. Alternatively, or additionally, a foggy area may havebeen identified (e.g., crowd-sourced data, weather data, etc.) and theprocess may be triggered when a location of the vehicle (e.g.,identified via GPS) enters or is in the identified foggy area. Inresponse to fog not being detected, the process can proceed to astandard imaging process. For example, the standard imaging processingcan include lighting without modulation (S715). In other words, thelight source doesn't need to be modulated to image a target object whenthere is no fog. Next, the detector 115 can acquire the images (S720),and the images can be displayed (and/or used for autonomous drivingoperations, for example) in S725. After the images are displayed inS725, the process can end. However, in response to fog being detected,the system 100 can image a target object through a scattering medium(e.g., fog) based on a fog module in S710. The target object can be anobject in or through the fog. For example, in one aspect, the apparatus102 can be a vehicle and the object in or through the fog can be anobject relevant to vehicle operation (e.g., another vehicle, a streetsign, lane lines, etc.). The operation of the fog module in S710 isfurther described in FIG. 7B.

FIG. 7B is an algorithmic flow chart of a method for imaging a targetobject through a scattering medium according to one or more aspects ofthe disclosed subject matter.

In S730, the system 100 (e.g., via the processing circuitry 130) canevaluate a transmission matrix (TM) or field distribution in lowdimension for one localized illumination. Evaluating the TM or fielddistribution in low dimension for one localized illumination can beaccomplished using the Hadamard matrix approach or the phase conjugationapproach. Steps S735-S770 further describe the evaluation of the TM orfield distribution in low dimension for one localized illumination inS730.

In S735, the processing circuitry 130 can determine if the Hadamardmatrix approach is fast. For example, it can be determined if theHadamard matrix approach can be calculated faster than a predeterminedthreshold. In other words, it can be determined if the Hadamard matrixapproach can be fast enough to calculate the TM in low dimension for onelocalized illumination faster than the decorrelation time while stillleaving enough time for the association technique. For example, if theHadamard matrix approach can calculate the TM in low dimension for onelocalized illumination in 1 ms, then the association technique furtherdescribed herein to image a target object through fog can be performedwithin the decorrelation time of fog (e.g., within 5 ms). If theHadamard matrix approach is fast (i.e., meets the predeterminedthreshold for the scattering medium), the process can continue to theHadamard matrix approach (S740-S755). Although the Hadamard matrixapproach needs to be fast to continue to the transmission matrixevaluation in this example, it should be appreciated that when theHadamard matrix approach is fast, the process can also continue to thephase conjugation approach (S760-S770). However, if the Hadamard matrixis not fast in S735 (i.e., does not meet the predetermined threshold forthe scattering medium), the process can continue to the phaseconjugation approach (S760-S770).

In S740, in the Hadamard matrix approach, the processing circuitry 130can generate a Hadamard matrix for the one localized illumination.

In S745, the processing circuitry 130 can input columns of the Hadamardmatrix on a modulator (e.g., the modulator 110).

In S750, the processing circuitry 130 can measure the intensities on thedetector 115.

In S755, the processing circuitry 130 can calculate the TM in lowdimension for one localized illumination based on the measuredintensities. It should be appreciated that steps S740-S755 are similarlydescribed in FIG. 3 in more detail. After the calculating the TM inS755, the process can continue to induce TM or field distributions inhigh dimension for the one localized illumination in S775.

Referring back to when it is determined that the Hadamard matrixapproach is not fast in S735, the process continues with the phaseconjugation approach. In S760, the processing circuitry 130 can locate aTM in a database (e.g., database 120). The TM stored in the database 120can have been calculated previously for one localized illuminationwithout fog, for example. In one aspect, the database 120 can storeinformation corresponding to a plurality of transmission matrixes whereeach transmission matrix indicates a linear input/output response toeach scattering medium of a plurality of scattering mediums.Additionally, the database 120 can store information regarding phasepatterns at the spatial light modulator for generating all of theplurality of localized illuminations. The database 120 can also storeinformation of phase difference at the spatial light modulator where thephase difference can be determined between a standard phase pattern forgenerating one of the plurality of localized illuminations and otherphase patterns for generating neighboring localized illuminations.

In S765, the processing circuitry 130 can apply the TM on the modulator110 and form one localized illumination with ballistic light.

In S770, the detector 115 can measure the optical phase (i.e.,transmission matrix) formed by the light reflected from the onelocalized illumination passing through the fog. In S771, the processingcircuitry 130 can induce the conjugation of the optical phase, as afield distribution in low dimension for one localized illumination.Then, the processing circuitry 130 can induce a field distribution inhigh dimension for the one localized illumination in S775.

In S775, the processing circuitry 130 can induce TM or fielddistribution in high dimension for the one localized illumination byusing the field distribution in low dimension for the one localizedillumination.

In S780, the processing circuitry 130 can induce field distributions(phase or amplitude) in high dimension for a set of localizedilluminations based on the induced field distribution for the onelocalized illumination in S775. More specifically, the fielddistributions for the set of localized illuminations can be based on anassociation relationship among the field distributions as has beenfurther described herein. For example, the neighboring localizedilluminations can be induced based on the one localized illumination inhigh dimension induced in S775. In other words, in one aspect, the phasepatterns for other localized illuminations of light neighboring the onelocalized illumination can be induced such that the phase patterns forthe neighboring localized illuminations can be identified. Additionally,when the phase patterns are induced, the processing circuitry 130 canfurther calculate the phase patterns based on a periodical ornon-periodical phase difference between phase patterns of mutuallyadjacent localized illuminations. Additionally, in one aspect, the phasepatterns can be abstracted from a plurality of phase patterns forgenerating the plurality of localized illuminations at the spatial lightmodulator. In other words, the one localized illumination can be astarting point and the calculation can be repeated to calculate the restof the phase patterns to identify the other neighboring localizedilluminations.

In S785, the processing circuitry 130 can apply field distributions onthe modulator, scanning the plurality of the focused points on thetarget object. In other words, the processing circuitry 130 can drivemodulator 110 via modulation signals for irradiating patterns forgenerating a plurality of localized illuminations from the light source105 arranged in a matrix on the target object.

In S790, the processing circuitry 130 can detect the images (e.g., viathe detector 115) of the plurality of localized illuminations of thetarget object based on the scanning in S785. The processing circuitry130 can further process the detected images to acquire a confocal imageof the target object. After acquiring the confocal image of the targetobject, the process can end.

The system 100 can include several advantages. For example, the overallscattering properties of the scattering medium can be established via alimited knowledge on the scattering processing. For example, anassociation relationship in scattering processing can be employed toacquire the optical field distribution to manipulate the scatteredlight. Additionally, both optical phase and amplitude can beindividually induced from the associations. The complex amplitude, whichis composed of its phase and amplitude, can also be induced from theoptical field association to manipulate the scattered light. As aresult, the association of the optical fields can be employed to revealthe scattering property of the scattering medium.

In the above description of FIGS. 2, 3, and 7, any processes,descriptions or blocks in flowcharts and workflows can be understood asrepresenting modules, segments or portions of code which include one ormore executable instructions for implementing specific logical functionsor steps in the process, and alternate implementations are includedwithin the scope of the exemplary embodiments of the presentadvancements in which functions can be executed out of order from thatshown or discussed, including substantially concurrently or in reverseorder, depending upon the functionality involved, as would be understoodby those skilled in the art. The various elements, features, andprocesses described herein may be used independently of one another ormay be combined in various ways. All possible combinations andsub-combinations are intended to fall within the scope of thisdisclosure.

Next, a hardware description of a computer/device according to exemplaryembodiments is described with reference to FIG. 8. The hardwaredescription described herein can be a hardware description of theprocessing circuitry 130, for example. In FIG. 8, the processingcircuitry 130 includes a CPU 800 which performs one or more of theprocesses described above/below. The process data and instructions maybe stored in memory 802. These processes and instructions may also bestored on a storage medium disk 804 such as a hard drive (HDD) orportable storage medium or may be stored remotely. Further, the claimedadvancements are not limited by the form of the computer-readable mediaon which the instructions of the inventive process are stored. Forexample, the instructions may be stored on CDs, DVDs, in FLASH memory,RAM, ROM, PROM, EPROM, EEPROM, hard disk or any other informationprocessing device with which the processing circuitry 130 communicates,such as a server or computer.

Further, the claimed advancements may be provided as a utilityapplication, background daemon, or component of an operating system, orcombination thereof, executing in conjunction with CPU 800 and anoperating system such as Microsoft Windows, UNIX, Solaris, LINUX, AppleMAC-OS and other systems known to those skilled in the art.

The hardware elements in order to achieve the processing circuitry 130may be realized by various circuitry elements. Further, each of thefunctions of the above described embodiments may be implemented bycircuitry, which includes one or more processing circuits. A processingcircuit includes a particularly programmed processor, for example,processor (CPU) 800, as shown in FIG. 8. A processing circuit alsoincludes devices such as an application specific integrated circuit(ASIC) and conventional circuit components arranged to perform therecited functions.

In FIG. 8, the processing circuitry 130 includes a CPU 800 whichperforms the processes described above. The processing circuitry 130 maybe a general-purpose computer or a particular, special-purpose machine.In one embodiment, the processing circuitry 130 becomes a particular,special-purpose machine when the processor 800 is programmed to performassociation assisted establishment of scattering configuration inscattering processing (and in particular, any of the processes discussedwith reference to FIGS. 2, 3, and 7).

Alternatively, or additionally, the CPU 800 may be implemented on anFPGA, ASIC, PLD or using discrete logic circuits, as one of ordinaryskill in the art would recognize. Further, CPU 800 may be implemented asmultiple processors cooperatively working in parallel to perform theinstructions of the inventive processes described above.

The processing circuitry 130 in FIG. 8 also includes a networkcontroller 806, such as an Intel Ethernet PRO network interface cardfrom Intel Corporation of America, for interfacing with network 828. Ascan be appreciated, the network 828 can be a public network, such as theInternet, or a private network such as an LAN or WAN network, or anycombination thereof and can also include PSTN or ISDN sub-networks. Thenetwork 828 can also be wired, such as an Ethernet network, or can bewireless such as a cellular network including EDGE, 3G and 4G wirelesscellular systems. The wireless network can also be WiFi, Bluetooth, orany other wireless form of communication that is known.

The processing circuitry 130 further includes a display controller 808,such as a graphics card or graphics adaptor for interfacing with display810, such as a monitor. A general purpose I/O interface 812 interfaceswith a keyboard and/or mouse 814 as well as a touch screen panel 816 onor separate from display 810. General purpose I/O interface alsoconnects to a variety of peripherals 818 including printers andscanners.

A sound controller 820 is also provided in the processing circuitry 130to interface with speakers/microphone 822 thereby providing soundsand/or music.

The general-purpose storage controller 824 connects the storage mediumdisk 804 with communication bus 826, which may be an ISA, EISA, VESA,PCI, or similar, for interconnecting all of the components of theprocessing circuitry 130. A description of the general features andfunctionality of the display 810, keyboard and/or mouse 814, as well asthe display controller 808, storage controller 824, network controller806, sound controller 820, and general purpose I/O interface 812 isomitted herein for brevity as these features are known.

The exemplary circuit elements described in the context of the presentdisclosure may be replaced with other elements and structureddifferently than the examples provided herein. Moreover, circuitryconfigured to perform features described herein may be implemented inmultiple circuit units (e.g., chips), or the features may be combined incircuitry on a single chipset.

The functions and features described herein may also be executed byvarious distributed components of a system. For example, one or moreprocessors may execute these system functions, wherein the processorsare distributed across multiple components communicating in a network.The distributed components may include one or more client and servermachines, which may share processing, in addition to various humaninterface and communication devices (e.g., display monitors, smartphones, tablets, personal digital assistants (PDAs)). The network may bea private network, such as a LAN or WAN, or may be a public network,such as the Internet. Input to the system may be received via directuser input and received remotely either in real-time or as a batchprocess. Additionally, some implementations may be performed on modulesor hardware not identical to those described. Accordingly, otherimplementations are within the scope that may be claimed.

Having now described embodiments of the disclosed subject matter, itshould be apparent to those skilled in the art that the foregoing ismerely illustrative and not limiting, having been presented by way ofexample only. Thus, although particular configurations have beendiscussed herein, other configurations can also be employed. Numerousmodifications and other embodiments (e.g., combinations, rearrangements,etc.) are enabled by the present disclosure and are within the scope ofone of ordinary skill in the art and are contemplated as falling withinthe scope of the disclosed subject matter and any equivalents thereto.Features of the disclosed embodiments can be combined, rearranged,omitted, etc., within the scope of the invention to produce additionalembodiments. Furthermore, certain features may sometimes be used toadvantage without a corresponding use of other features. Accordingly,Applicant(s) intend(s) to embrace all such alternatives, modifications,equivalents, and variations that are within the spirit and scope of thedisclosed subject matter.

1. A system, comprising: a spatial light modulator configured tomodulate one or more of phase and amplitude of light irradiated from alight source to a target object; and processing circuitry configured toevaluate a field distribution for one localized illumination, induce aset of field distributions for a plurality of localized illuminationsbased on the field distribution for the one localized illumination, andapply the set of field distributions to the spatial light modulator,scanning a plurality of localized illuminations on the target object. 2.The system of claim 1, wherein the processing circuitry is furtherconfigured to induce the field distribution from low dimension to highdimension for the one localized illumination.
 3. The system of claim 1,wherein the processing circuitry is further configured to detect animage for each of the plurality of the localized illuminations on thetarget object, wherein the images are processed to acquire a confocalimage of the target object.
 4. The system of claim 1, wherein theprocessing circuitry for evaluating the field distribution for the onelocalized illumination is further configured to determine whether aHadamard matrix approach satisfies a predetermined threshold requiredfor calculating the field distribution for the one localizedillumination, and calculate the field distribution for the one localizedillumination in response to the Hadamard matrix approach satisfying thepredetermined threshold.
 5. The system of claim 1, wherein periodicalphase differences corresponding to the set of field distributions areabstracted from a plurality of phase patterns for generating theplurality of localized illuminations at the spatial light modulator. 6.The system of claim 1, further comprising: a memory configured to storeinformation including a plurality of field distributions where eachfield distribution indicates a linear input/output response to eachscattering medium of a plurality of scattering mediums, phase patternsat the spatial light modulator for generating all of the plurality oflocalized illuminations, phase difference at the spatial light modulatorwhere the phase difference can be determined between a standard phasepattern for generating one of the plurality of localized illuminations,and other phase patterns for generating neighboring localizedilluminations.
 7. The system of claim 4, wherein the processingcircuitry for calculating the field distribution for one localizedillumination is further configured to generate a Hadamard matrix, inputcolumns from the Hadamard matrix on the spatial light modulator, measureintensities on the detector corresponding to the input on the spatiallight modulator, and calculate the field distribution for the onelocalized illumination based on the measured intensities.
 8. The systemof claim 1, wherein the processing circuitry is further configured tocalculate a phase or amplitude pattern based on the field distributionfor the one localized illumination, and induce a set of phase oramplitude patterns based on the calculated phase or amplitude pattern.9. The system of claim 8, wherein in response to inducing the phase andamplitude patterns based on the calculated phase pattern, the processingcircuitry is further configured to calculate one of the phase oramplitude patterns corresponding to a neighboring localized illuminationwhich is adjacent to the one localized illumination.
 10. The system ofclaim 1, wherein one of the plurality of localized illuminations islocated in a center of the plurality of localized illuminations arrangedon the target object.
 11. The system of claim 1, wherein the spatiallight modulator is a Digital Micromirror Device (DMD) including aplurality of micromirrors arranged in a matrix.
 12. The system of claim1, further comprising: an imaging device configured to generate imagedata by capturing an image of the target object through a scatteringmedium.
 13. A method for generating a modulation signal for scatteringprocessing, comprising: evaluating a field distribution for onelocalized illumination; inducing a set of field distributions for aplurality of localized illuminations based on the field distribution forthe one localized illumination; and applying the set of fielddistributions to the spatial light modulator, scanning a plurality oflocalized illuminations on the target object.
 14. The method of claim13, further comprising: inducing the field distribution from lowdimension to high dimension for the one localized illumination.
 15. Themethod of claim 13, further comprising: detect an image for each of theplurality of the localized illuminations on the target object, whereinthe images are processed to acquire a confocal image of the targetobject.
 16. The method of claim 13, further comprising: determiningwhether a Hadamard matrix approach satisfies a predetermined thresholdrequired for calculating the field distribution for the one localizedillumination; and calculating the field distribution for the onelocalized illumination in response to the Hadamard matrix approachsatisfying the predetermined threshold.
 17. The method of claim 16,wherein calculating the field distribution for the one localizedillumination further comprises: generating a Hadamard matrix; inputtingcolumns from the Hadamard matrix on the spatial light modulator;measuring intensities on the detector corresponding to the input on thespatial light modulator; and calculating the field distribution for theone localized illumination based on the measured intensities.
 18. Themethod of claim 13, wherein one of the plurality of localizedilluminations is located in a center of the plurality of localizedilluminations arranged on the target object.
 19. The method of claim 13,wherein periodical phase differences between phase patternscorresponding to the set of field distributions are abstracted from aplurality of phase patterns for generating the plurality of localizedilluminations at the spatial light modulator.
 20. A system, comprising:a light source; a spatial light modulator; a detector; and processingcircuitry configured to project, via the light source, illuminationbased on a predetermined illumination profile, modulate, via the spatiallight modulator, an optical field of the projected illumination based onthe illumination profile, detect, via the detector, backscatteredillumination corresponding to a first portion of the projectedillumination backscattered from a scattering medium, detect, via thedetector, reflected illumination corresponding to a second portion ofthe projected illumination reflected from a target object located in orthrough the scattering medium, and induce the illumination profile basedon one localized illumination, wherein other localized illuminations canbe formed at different positions through the scattering medium based onan association relationship among the optical fields for the localizedilluminations in or through the scattering medium.